The present invention relates to a signal decoding method, a signal decoding apparatus, a signal multiplexing method, a signal multiplexing apparatus, and a recording medium and, more particularly to an apparatus and a method of implementing decoding and multiplexing in which amount of signals to be decoded can be reduced with effects on a regenerated image effectively suppressed, and an apparatus and a method in which multiplexed signals are decoded properly.
The present invention also relates to a recording medium which stores the multiplexed signals and a recording medium which stores a program for implementing the decoding and the multiplexing in a computer.
In order to record and transmit large volumes of image data with efficiency, techniques for compressing the image data are extremely important. So far, as the techniques, information compressive techniques according to standards such as JPEG (Joint Photographic Coding Experts Group), MPEG(Moving Picture Experts Group), and so forth have been developed. Image data of an image or a moving picture series is handled in MPEG and JPEG.
There is an image termed xe2x80x9ca composite imagexe2x80x9d. The xe2x80x9ccomposite imagexe2x80x9d comprises plural small images (object images). FIG. 8 is a diagram showing a concept of this composite image.
Referring now to FIG. 8, a final composite image 707 comprises a background image 701, and images 702 to 705 of objects A and B located in front of the background image 701. Note that a background image of a composite image is also an image of an object.
Specifically, the background image 701 and the image 702 of the object A are composited to generate a composite image 706. At this time, a transparency signal indicating a composition ratio of a pixel value of the background image 701 to a pixel value of the image 702, is used. The image 703 is an image represented by the transparency signal of the object A, which indicates transparency of the image of the object A.
Further, the composite image 706 and the image 704 are composited on the basis of a transparency signal of the object B as mentioned above, to generate the composite image 707. An image 705 is an image of the object B represented by the transparency signal of the object B, which indicates transparency of the image of the object B.
In a case where a moving picture is a composite image, a background image and an image of each object thereof are respectively assumed to be a moving picture series. Therefore, images of frames of each moving picture series are composited, to generate a composite moving picture comprising plural moving picture series.
FIG. 9 is a block diagram showing a prior art image multiplexing apparatus which codes and multiplexes image signals of a composite image and an image decoding apparatus which decodes multiplexed coded signals of the composite image.
Referring now to FIG. 9, an image multiplexing apparatus 801 is used for coding and multiplexing image signals (digital image data) of respective objects of the composite image, and an image decoding apparatus 802 is used for receiving and decoding multiplexed signals MEg output from the image multiplexing apparatus 801.
More specifically, the image multiplexing apparatus 801 comprises plural coding means 40a1 to 40an for coding image signals Sg1 to Sgn of plural objects of the composite image, to produce coded signals Eg1 to Egn, respectively, and multiplexing means 412 for multiplexing the coded signals Eg1 to Egn to produce multiplexed coded signals MEg.
In the image multiplexing apparatus 801 so constructed, when the image signals Sg1 to Sgn are input, the coding means 40a1 to 40an code the image signals Sg1 to Sgn, respectively, and outputs the coded signals Eg1 to Egn to the multiplexing means 412, which multiplexes the coded signals Eg1 to Egn and outputs the multiplexed signals MEg to a transmission path 803.
Meanwhile, the image decoding apparatus 802 comprises demultiplexing means 102 for demultiplexing the multiplexed signals MEg to separate the coded signals Eg1 to Egn from the multiplexed signals MEg, a decoding unit 101 for sequentially decoding the coded signals Eg1 to Egn in accordance with a control signal C1, synthesizing means 110 for receiving decoded signals Dg of respective objects output from the decoding unit 101 and synthesizing the decoded signals Dg in accordance with a control signals C2 to produce synthesized reproduced signals Rs, and a CPU 108 for producing the control signals C1 and C2.
The decoding unit 101 comprises a select switch 103 for selecting the separated coded signals Eg1 to Egn in the order in accordance with the control signal C1, and decoding means 109 for decoding a selected output Se of the select switch 103 to produce the decoded signal Dg. The CPU 108 has a capability of controlling switching of the select switch 103 by the control signal C1, and instructing the synthesizing means 110 on a synthesizing method by the control signal C2.
In the image decoding apparatus 802 so constructed, when the multiplexed signals MEg output from the image multiplexing apparatus 801 are input through the transmission path 803, the multiplexed signals MEg are demultiplexed by the demultiplexing means 102 to produce the coded signals Eg1 to Egn.
At this time, the CPU 108 produces the control signals C1 and C2. The select switch 103 selects one of the coded signals Eg1 to Egn in accordance with the control signal C1 and outputs the selected signal Se to the decoding means 109, which decodes the selected signal Se and outputs the decoded signal Dg to the synthesizing means 110, which synthesizes the decoded signals Dg in accordance with the control signal C2 and, outputs the synthesized reproduced signals Rs.
A case of processing the composite moving picture comprising plural small images using these apparatus will be described. When image signals of moving picture series of respective objects are input to the image multiplexing apparatus 801, the coding means 40a1 to 40an code the image signals, respectively, and the multiplexing means 412 multiplexes these coded signals and outputs the multiplexed signals MEg to the image decoding apparatus 802. In the image decoding apparatus 802, the demultiplexing means 102 separates the coded signals Eg1 to Egn of respective objects of the moving picture series from the multiplexed signals MEg, the decoding unit 101 decodes these coded signals, and the synthesizing means 110 synthesizes the decoded signals Dg and outputs the synthesized reproduced signals Rs of the composite moving picture comprising the moving picture.
Note that in the image multiplexing apparatus 801, information required for compositing images, such as compositing order, size, and compositing positions of images of respective objects is also multiplexed and transmitted during multiplexing the coded signals.
However, since the composite image comprises plural images, load on decoding is increased as compared with a case where an image is decoded. For this reason, in decoding in software using a general purpose computer, the size and number of images to be decoded within a given time are limited depending on capability of the CPU. In addition, in a case where the composite image comprises a moving picture, although coded signals of a prescribed amount must be decoded within a given time, if the decoding is not completed within the given time, display delay or overflow of an input buffer occurs.
As should be appreciated from the forgoing, it is important that decoding be controlled depending on capability of a decoding apparatus when the coded signals of the composite image are decoded. However, in the prior art, it is difficult to control the decoding, since insufficient capability of the decoding apparatus is not taken into account.
It is an object of the present invention to provide a method and an apparatus of multiplexing images in which decoding is performed depending on capability of a decoding apparatus, and a method and an apparatus of decoding an image in which amount of signals to be decoded can be reduced depending on decoding capability with adverse effects on regenerated images suppressed.
It is another object of the present invention to provide a recording medium which stores multiplexed signals of data structure allowing decoding depending on capability of the decoding apparatus and, a recording medium which stores a program for implementing signal processing with the image multiplexing method and the image decoding method in a computer.
Other objects and advantages of the invention will become apparent from the detailed description that follows. The detailed description and specific embodiments described are provided only for illustration since various additions and modifications within the spirit and scope of the invention will be apparent to those of skill in the art from the detailed description.
According to a first aspect of the present invention, a signal decoding method comprising the steps of: receiving multiplexed signals in which coded signals of plural pieces of information to be recorded or transmitted have been multiplexed; obtaining priority information of respective information from the multiplexed signals, and decoding coded signals of respective information included in the multiplexed signals in the order in accordance with the obtained priority information.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, decoding can be adaptively controlled so as to minimize adverse effects on regenerated images.
According to a second aspect of the present invention, a signal decoding comprises the steps of: receiving multiplexed signals in which coded signals of plural pieces of information to be recorded or transmitted have been multiplexed; obtaining priority information of respective information from the multiplexed signals; deciding the order in which the coded signals are decoded in accordance with the obtained priority information; and decoding only coded signals of a higher decoding order than a prescribed order, of the coded signals of plural pieces of information included in the multiplexed signals.
Therefore, if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, only coded signals of significant information can be decoded.
According to a third aspect of the present invention, a signal decoding method comprises the steps of: receiving multiplexed signals in which hierarchically coded signals comprising coded signals in plural hierarchies of plural pieces of information to be recorded or transmitted have been multiplexed; obtaining priority information of respective information from the multiplexed signals; deciding the order in which the hierarchically coded signals are decoded in accordance with the obtained priority information; and decoding only hierarchically coded signals in hierarchies from the lowest order to a higher order than a prescribed order, of a higher decoding order than a prescribed order, of the hierarchically coded signals of plural pieces of information included in the multiplexed signals.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding hierarchically coded signals of all objects included in the multiplexed signals, decoding can be adaptively controlled so as to minimize adverse effects on regenerated images
According to a fourth aspect of the present invention, the signal decoding method of the first aspect further comprises the steps of: detecting amounts of the coded signals of respective information included in the multiplexed signals; and deciding the priority information such that the priority decreases in decreasing order of amounts of coded signals.
Therefore, if processing capability of a decoding apparatus is not sufficient for decoding the multiplexed signals, amount of signals to be decoded can be reduced while minimizing adverse effects on regenerated images.
According to a fifth aspect of the present invention, in the signal decoding method of the first aspect, respective information to be recorded or transmitted comprises digital image data as digital data with which an image is displayed, and the multiplexed signals comprise a coded image signal in which the digital image data has been coded, as the coded signals.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, decoding can be adaptively controlled so as to minimize adverse effects on regenerated images.
According to a sixth aspect of the present invention, the signal decoding method of the fifth aspect further comprises the steps of: detecting size of regenerated images in which the coded image signals of the multiplexed signals have been decoded; and deciding the priority information such that the priority of the digital image data decreases in decreasing order of size of the regenerated images.
Therefore, if processing capability of a decoding apparatus is not sufficient for decoding the multiplexed signals, amount of signals to be decoded can be reduced while minimizing adverse effects on regenerated images.
According to a seventh aspect of the present invention, the signal decoding method of the fifth aspect further comprises the steps of: detecting overlapping order in which regenerated images in which the coded image signals of the multiplexed signals have been decoded are composited; and deciding the priority information such that a higher priority is assigned to digital image data of a regenerated image located in front in relation to the other regenerated images, in a composite image comprising the overlapped regenerated images.
Therefore, if processing capability of a decoding apparatus is not sufficient for decoding the multiplexed signals, amount of signals to be decoded can be reduced while minimizing adverse effects on regenerated images.
According to an eighth aspect of the present invention, the signal decoding method of the fifth aspect further comprises the steps of: deciding whether a regenerated image in which a coded image signal of the multiplexed signals has been decoded is used as a reference image for decoding another coded image signal or not; and deciding the priority information such that a higher priority is assigned to digital image data of a regenerated image which is used as the reference image for the decoding, in relation to the other digital image data of regenerated images which are not used as the reference image for the decoding.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, amount of signals to be decoded can be reduced, while preferentially decoding a coded signal of a reference image required for predictive coding.
According to a ninth aspect of the present invention, the signal decoding method of the fifth aspect further comprises the steps of: deciding whether the coded image signal of the digital image data is an intra frame coded image signal or not; and deciding the priority information such that a higher priority is assigned to digital image data of the intra frame coded image signal in relation to the other digital image data in which coded image signals are not intra frame coded image signals.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, amount of signals to be decoded can be reduced, while preferentially decoding a coded signal which plays an important role in high-speed retrieval.
According to a tenth aspect of the present invention, a signal decoding apparatus which decodes multiplexed signals in which coded signals of plural pieces of information to be recorded or transmitted, have been multiplexed, comprises: demultiplexing means for separating coded signals of respective information from the multiplexed signals; priority producing means for producing priority information of respective information on the basis of the multiplexed signals; decoding means for decoding the coded signals output from the demultiplexing means in a prescribed order in accordance with a control signal; and control means for controlling the order in which coded signals are decoded by the decoding means, by the control signal on the basis of the priority information.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, decoding can be adaptively controlled so as to minimize adverse effects on regenerated images.
According to an eleventh aspect of the present invention, a signal multiplexing method comprising the steps of: coding plural pieces of information to be recorded or transmitted, to produce coded signals; producing priority information of respective information with which decoding order of the coded signals is decided, on the basis of the coded signals; and multiplexing the coded signals together with the priority information.
Therefore, it is not necessary for an image decoding apparatus to set priorities of respective information, whereby burden of signal processing on the image decoding apparatus can reduced.
According to a twelfth aspect of the present invention, in the signal multiplexing method of the eleventh aspect, the priority information of respective information is decided such that the priority decreases in decreasing order of amounts of coded signals of respective information.
Therefore, if processing capability of a decoding apparatus is not sufficient for decoding the multiplexed signals, amount of signals to be decoded can be reduced while minimizing adverse effects on regenerated images.
According to a thirteenth aspect of the present invention, in the signal multiplexing method of the eleventh aspect, respective information to be recorded or transmitted comprises digital image data as digital data with which an image is displayed, and the multiplexed signals comprise a coded image signal in which the digital image data has been coded, as the coded signals.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, decoding can be adaptively controlled, while minimizing adverse effects on regenerated images.
According to a fourteenth aspect of the present invention, in the signal multiplexing method of the thirteenth aspect, the priority information of each digital image data is decided on the basis of size of regenerated images in which coded image signals of the multiplexed signals have been decoded, and decided such that the priority of digital image data decreases in decreasing order of size of the regenerated images.
Therefore, if processing capability of a decoding apparatus is not sufficient for decoding the input multiplexed signals, amount of signals to be decoded can be reduced while minimizing adverse effects on regenerated images.
According to a fifteenth aspect of the present invention, in the signal multiplexing method of the thirteenth aspect, the priority information of each digital image data indicates the overlapping order in which regenerated images in which coded image signals of the multiplexed signals have been decoded are composited, and the priority information is decided such that a higher priority is assigned to digital image data of a regenerated image located in front in relation to the other regenerated images, in a composite image comprising the overlapped regenerated images.
Therefore, if processing capability of a decoding apparatus is not sufficient for decoding the input multiplexed signals, amount of signals to be decoded can be reduced while minimizing adverse effects on regenerated images.
According to a sixteenth aspect of the present invention, in the signal multiplexing method of the thirteenth aspect, the priority information of each digital image data is decided based on whether a regenerated image in which a coded image signal of the multiplexed signals has been decoded is used as a reference image for decoding another coded image signal or not, and decided such that a higher priority is assigned to digital image data of a regenerated image which is used as the reference image for the decoding, in relation to the other regenerated images which are not used as the reference image for the decoding.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding the input multiplexed coded signals, amount of signals to be decoded can be reduced, while preferentially decoding a coded signal of a reference image required for predictive coding.
According to a seventeenth aspect of the present invention, in the signal multiplexing method of the thirteenth aspect, the priority information of the digital image data is decided based on whether a coded image signal of each digital image data of the multiplexed signals is an intra-frame coded signal or not, and decided such that a higher priority is assigned to digital image data of the intra frame coded signal, in relation to the other digital image data in which coded image signals are not intra frame coded image signals.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding the input multiplexed signals, amount of coded signals to be decoded can be reduced, while preferenlially decoding a coded signal which plays an important role in high-speed retrieval.
According to an eighteenth aspect of the present invention, a signal multiplexing apparatus which multiplexes coded signals of plural pieces of information to be recorded or transmitted, comprises priority producing means for producing priority information of respective information with which decoding order of the coded signals is decided; and multiplexing means for multiplexing the coded signals together with the priority information.
Therefore, it is not necessary for an image decoding apparatus to set priorities of respective information, whereby burden of signal processing on the image decoding apparatus can reduced.
According to a nineteenth aspect of the present invention, a recording medium is used for storing multiplexed signals in which coded signals of plural pieces of information to be recorded or transmitted have been multiplexed together with priority information of respective information with which decoding order of the coded signals is decided.
Therefore, in the decoding apparatus which decodes the multiplexed signals, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, decoding can be adaptively controlled, while minimizing adverse effects on regenerated images.
According to a twentieth aspect of the present invention, a recording medium is used for storing a program which makes a computer perform signal decoding, the program making the computer perform signal decoding by the signal decoding method of the first aspect.
Therefore, even if signal processing capability of a decoding apparatus is not sufficient for decoding all the multiplexed coded signals, adaptively controlled decoding while minimizing adverse effects on regenerated images is realized in the computer.
According to a twenty-first aspect of the present invention, a recording medium is used for storing a program which makes a computer perform signal multiplexing, the program making the computer perform signal multiplexing by the signal multiplexing method of the eleventh aspect.
Therefore, it is not necessary for a signal decoding apparatus to set priorities of respective information, whereby signal multiplexing in which processing burden on the signal decoding apparatus is reduced, is realized in the computer